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Abstract 



Let G be a finite strongly connected aperiodic directed graph in which each edge 
carries a label from a finite alphabet A. Then G induces a trellis coded quantizer 
for encoding an alphabet A memory less source. A source sequence of long finite 
length is encoded by finding a path in G of that length whose sequence of labels is 
closest in Hamming distance to the source sequence; finding the minimum distance 
path is a dynamic programming problem that is solved using the Viterbi algorithm. 
We show how a Markov chain can be used to obtain a closed form expression for the 
asymptotic expected Hamming distortion per sample that results as the number of 
encoded source samples increases without bound. 

1 Introduction 

Let G be a finite strongly connected aperiodic directed graph. We further suppose that 
G is a labelled graph in the sense that each edge e of G carries a label L(e) from a fixed 
finite alphabet A. 

Let d : A x A — >■ {0, 1} be the Hamming distance function 



Let {X,}- x be an i.i.d. stochastic process with alphabet A. In this paper, we develop a 
method to compute the number 



where n n = (ei,e 2 , • • • , e n ) varies over all paths of length n in G, consisting of edges 
ei, C2-, ■ ■ ■ , e n . The limit in ([1]) exists by the subadditive ergodic theorem. 

We discuss why it is important to know how to compute D(G). Let R be a positive 
integer, and suppose G is chosen to have 2 R outgoing edges per vertex. The labelled graph 
G induces a trellis coded quantizer [T] for encoding the samples Xi,X%, ■ ■ • ,X n for any 
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positive integer n. The induced trellis coded quantizer operates in two steps. In the first 
step, the Viterbi algorithm is used to find an optimal path (ei, e 2 , • • • , e n ) in G along which 
the total distortion Y^=i d(Xi, L(e.i)) is minimized; the sequence (L(ei), L(e 2 ), • ■ ■ , L(e n )) 
obtained by following the optimal path (ei, e 2 , ■ ■ ■ , e n ) is consequently the best quantized 
version of the sequence (Xi,X 2 , ■ ■ ■ , X n ). In the second step, the trellis coded quantizer 
encodes the optimal path into 

B n = nk+ [log 2 card(V G y\ 

bits, where V G denotes the set of vertices of G and card(V G ) denotes the cardinal- 
ity of the set V G ; these B n bits are transmitted to the user. From the B n received 
bits, the user determines the optimal path (ei, e 2 , ■ ■ • , e n ) and then builds the sequence 
(L(ei), L(e 2 ), • • - , L(e n )) by following this path; by construction, it is guaranteed that 
the sequence (L(ei), L(e 2 ), ■ • • , L(e n )) is the user's best possible estimate of the sequence 
(Xi,X 2 , • • ■ ,X n ). The asymptotic compression rate achieved by the trellis coded quan- 
tizer as n — > oo is R bits/sample, since 

R = lim n~ l B n . 

n— >oo 

Equation (JTJ tells us that D(G) is the asymptotic expected distortion/sample generated 
by the trellis coded quantizer as n — > oo. The information source with output 
has a distortion-rate function which may be computed via the Blahut algorithm; let D(R) 
be the distortion-rate function evaluated at rate R. Then, D(G) > D(R) and the trellis 
coded quantizer induced by G performs well if D(G) « D(R). It is an unsolved problem 
of source coding theory to find in a fast manner a labelled graph G with 2 R outgoing 
edges per vertex for which D(G) is as close as desired to D(R). Since this paper allows 
one to explicitly compute D(G) for a fixed labelled graph G, it is hoped that progress 
can be made in the search for a G for which D(G) ~ D(R) by examining how D{G) 
varies as one varies G. 

2 The Method 

The Viterbi algorithm can be used to compute the minimum over Tr n in (pQ); we describe 
how this is done. For each vertex v of G, let T{v) be the set of all pairs (v r , e) such that 
v' is a vertex of G, e is an edge of G, and e leads from v' to v. Let S be the set of all 
vectors 

s = (s(v) : v G Vg) 

with nonnegative integer components. We define a "Viterbi transition operator" V : 
S x A —7- S as follows: For each s e S and x G A, define V(s, x) to be the vector 
s i — ( s i( v ) '■ v Vg) in S in which 

Si(v) = min{(V, e) G X{v) : s(v') + d(x, L(e))}, v G V G . 

For each i = 0, 1, 2, • • • , n, let S l be the random vector 

S l = (S>) : f G Vg) 

in which 5*° is the zero vector and 

S i = V{S*-\X i ), i = l,2,...,n. 
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The Viterbi algorithm tells us that 

n 

min S n = min{£ d(X h L(e,))}, (2) 

1=1 

where minS™ denotes the minimum component of vector S n . Notice that 
min S^ 1 < min S i < 1 + min ST' 1 , z = 1, 2, • • • , n, 

and therefore 

minS n = carrf({l <i<n:mmS i ^ minS^ 1 }). (3) 
For i = 0, 1, • • • , n, let 

S l = :t;eV G ) 

be the vector in which 

S i (v) = S i (v)-minS i , veV G . 

Let V : 5 x A — > S be the "reduced Viterbi transition operator" in which V(s, x) 
is obtained from V(s,x) by subtracting the minimum component of V(s,x) from each 
component of V(s, x). Then S°, S 1 , ■ ■ ■ , S n are obtained from the recursion 

S° = zero vector in S 

S { = V(^- 1 ,X J ), i=l,2,...,n (4) 

Let iSg be the set of all s in S in which s is the zero vector or in which there exist 
Xi, x 2 , ■ ■ ■ , x n G A for some positive integer n such that s is obtained from the recursion 

s° = zero vector in S 

= VXaT 1 ,^), i = l,2,...,n 
s n = s 

For each i = 1, 2, • • • , n, 

min 5* ^ min^ 1 ^ V(5 i - 1 ,X J ) ^ 5 G . 

This fact, coupled with ([3]), allows us to conclude that 

min S n = card({l <i<n: V^' 1 , Xi) <£ S G }). (5) 

Lemma 1 Sq is finite. Moreover, let k be the smallest positive integer such that it is 
possible to go from any vertex of G to any vertex of G along a path of length k. Then, if 
s G Sq, each component of vector s is <k. 

Proof. Let s G Sq- If s is the zero vector, we are done. Assume s is not the zero 
vector. We can therefore find Xi, ■ ■ ■ , x n in A for some positive integer n such that if 
5° is the zero vector in S and 

S i = V(s i ~\x i ), z = l,2,---,n, 

then s n = s. Let s°, s l , ■ ■ ■ , s n be the vectors in which s° = s° and 

s { = V(s i -\x i ), z = l,2,-..,n. 

3 



We have 

s l = s l -mms\ i = 1,2, - - • ,n. (6) 

We show that the maximum and minimum components of s n differ by at most k, which 
by (J6]) will complete the proof. (The maximum and minimum element of s n = s will 
then differ by at most k; since the minimum element of s is 0, the maximum element 
will be at most k.) If n < k, this is obvious because each component of s 11 is a sum of n 
Hamming distances. Assume n > k. Let m be the minimum component of s n and let M 
be the maximum component. There is a path (e[, e 2 , • • • , e' n ) in G such that 

n 

J2d(xi,L(e'^) =m. 

1=1 

Suppose v is the vertex of G such that the maximum component of s n is s n (v). By 
changing the last k edges of the path (e'±, e' 2 , • • • , e' n ), we can find a path (e'/, e 2 , ■ ■ • , e") 
in G ending at v ; this path automatically yields 

n 

Y,d{xi,He'!))<m + k. 
i=i 

The number M is the smallest sum 

n 

Y^d{x h L{ei)) 
i=i 

along all paths (ei, e2, ■ • • , e n ) in G ending at v. Therefore, M < m + k. This completes 
the proof of the lemma. 

We are now able to describe our method for computing D(G). For each x G A, let 
p(x) denote the "source letter probability" Pr[Xj — x]. If si, s 2 belong to the state space 
Sq, a "Markov transition" Si — )■ s 2 is defined if and only if V(si,x) = s 2 for some x G A. 

Theorem 1 There is a probability distribution (q(s) : s G <Sg) such that 

q(s') = £ g(a)p(rr), a' G 5 G ; (7) 

{{s,x)ES G xA:V(s,x)=s'} 

D(G) = Yl Q( s M x )- ( 8 ) 

{(s,x)g5 G x A: V r (s,x)^5 G } 

Furthermore, if the Markov transition relation — > on Sq has only one closed class of 
irreducible states, then the probability distribution (q(s)) satisfying Fty is unique. 

Proof. From the fact that Sg is finite and equations ([T]) (J2J) (jlj) (j5J), we can extract 
a limit to conclude that there exists a Sc-valued random variable S° and an A-valued 
random variable X± such that 

(i) Xi and X\ have the same distribution (p(x) : x G A). 

(ii) S* and Xi are independent. 
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(iii) Letting S 1 be the <5>G-valued random variable defined by 

S 1 = ViS ,^), 
then 5° and S 1 have the same distribution. 

(iv) D{G) = Pt[V{S°,X 1 )^S g }. 
By property (ii), 

PrfS 1 = a']= Ft IS° = s] Pv[V(s, X x ) = s'\. (9) 

ses G 

Let (q(s) : s G So) be the common probability distribution of S° and S 1 (guaranteed by 
property (iii)). Equation becomes if one makes the substitutions 

Prists'] = q(s'), 
Pr[5° = s] = q{s), 
Pr[V(s,X l ) = s'] = P( x )- 

{xeA:V(s,x)=s'} 

(The third of these equations follows from property (i).) From properties (ii),(iv), 

D(G) = Pr[y(5°,X 1 ) £ S G ] = J2 p r[S° = a] Pr[F( S ,X 1 ) £ S G ]. 

ses G 

This equation reduces to (JHJ). The last part of Theorem 1 concerning the uniqueness of 
(q(s)) is a well-known fact from Markov chain theory. 



3 Example: de Bruijn graph 

We take G to be the labelled de Bruijn graph of size 8 having 2 incoming and 2 outgoing 
edges per vertex, depicted in trellis form in the figure below; the edge labels are chosen 
from the alphabet A = {a, b, c, d} of size 4. 
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We take our source output {Xi} to consist of independent equiprobable A- valued 
random variables. Let us compute D(G). 

The state space Sq consists of 107 vectors of length 8, which were found by computer 
search. It became convenient to partition these 107 vectors as follows: 
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= {00000000} 

52 = {00001111, 11110000} 

53 = {01101111, 10011111, 11110110, 11111001} 

£4 = {11101122, 11011122, 22110111, 22111011, 01112211, 10112211, 11221110, 11221101} 

s 5 = {00001100, 00000011, 00110000, 11000000} 

S 6 = {22221210, 22222101, 22220121, 22221012, 10122222, 01212222, 21012222, 12102222} 

s 7 = {10111100, 01111100, 11010011, 11100011, 00111101, 00111110, 11001011, 11000111} 

S s = {22332101, 22331210, 33221012, 33220121, 10123322, 01213322, 21012233, 12102233} 

Sg = {11101111, 11011111, 01111111, 10111111, 11110111, 11111011, 11111110, 11111101} 

510 = {22111001, 22110110, 11221001, 11220110, 10011122, 01101122, 10012211, 01102211} 

511 = {22221110, 22221101, 22220111, 22221011, 10112222, 01112222, 11012222, 11102222} 

512 = {21011122, 12101122, 10122211, 01212211, 22111012, 22110121, 11222101, 11221210} 
s ri = {10011100, 01101100, 10010011, 01100011, 00111001, 00110110, 11001001, 11000110} 

514 = {10000011, 01000011, 00111000, 00110100, 11000001, 00011100, 00101100, 11000010} 

515 = {22110001, 00012211, 11220100, 01001122, 22110010, 00102211, 11221000, 10001122} 
Sie = {10001111, 01001111, 11111000, 11110100, 11110010, 11110001, 00011111, 00101111} 

We call the sets Si,S 2 ,---, S 16 "superstates". They are arrived at as follows. Let 

11 = {7Ti, 7T 2 , 7T 3 , 7T 4 , 7T 5 , 7T 6 , 7T 7 , 7T 8 } 

be the group (under composition) of permutations of S defined by 

■K 1 (n 1 ,n 2 ,n 3 ,n4,n 5 ,n 6 ,n 7 ,n 8 ) = (ni, n 2 , n 3 , n A , n 5 , n 6 , n 7 , n 8 ) 

^2(ni,n 2 ,n 3 ,n4,n 5 ,n 6 ,n7,n 8 ) = (n 2 , ni, n 4 , n 3 , n 6 , n 5 , n 8 , n 7 ) 

^3(ni,n 2 ,n 3 ,n 4 ,n 5 ,n 6 ,n 7 ,n 8 ) = (n 3 , n 4 , n u n 2 , n 7 , n 8 , n 5 , n 6 ) 

^4{ni,n 2 ,n 3 ,n4,n 5 ,n 6 ,n 7 ,ns) = {n 4 ,n 3 ,n 2 ,n 1 ,n s ,n 7 ,n 6 ,n 5 ) 

n 5 (ni,n 2 ,n 3 ,ri4,n 5 ,n 6 ,n 7 ,n 8 ) = (n 5 ,n 6 ,n 7 ,n 8 ,n 1 ,n 2 ,n 3 ,n4) 

7r 6 (ni,n 2 ,n3,n4,n 5 ,n 6 ,n7,n 8 ) = (n^n b ,n 8 ,n 7 ,n 2 ,ni,n^n 3 ) 

7T 7 (ni,n 2 ,n 3 ,n4,n 5 ,n 6 ,n 7 ,n 8 ) = (n 7 , n 8 , n 5 , n 6 , n 3 , n 4 , ni, n 2 ) 

n 8 (n 1 ,n 2 ,n 3 ,n4,n 5 ,n 6 ,n 7 ,n 8 ) = (ns,n 7 ,n 6 ,n 5 ,n 4 ,n 3 ,n 2 ,n l ) 

If s G S, let Us denote the set 

Us = {tt(s) : tt G n}. 

The sets lis, s G S, are called II- fibers. Any two II- fibers either coincide or have empty 
intersection. The state space So is a union of Il-fibers; these Il-fibers are the superstates 
Si, S 2 , ■ ■ ■ , S\q. We let iS|II denote the set of all Il-fibers. We let iSg|I1 denote the set of 
all LT-fibers which are subsets of Sq] the set Sg\U is the same as the set of superstates 
{Si, S 2 , • • • , Siq}. 

For any subgroup IT of II, we can define the notion of Il'-fiber in a similar fashion. 
Let IT be the following subgroup of II: 

IT = {7ri,7r 2 ,7r 5 ,7r 6 }. 
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Let S G S G \U be any superstate. Let S 1 and S 2 be LT'-fibers whose union is S. The 
following are true: 

(a) The set 

{V(s, x) : (s, x) G S l x {a, 6}} U {V(s, x) : (s, x) G S* 2 x {c, rf}} (10) 
is a LT-fiber. 

(b) The set 

{V(s, x) : (s, x) G S 2 x {a, 6}} U {V(s, x) : (s, i) 6 S 1 x {c, rf}} (11) 
is a LT-fiber. 

Define f/(S, 0) to be the IT-fiber (P3JJ) and define [/(£, 1) to be the n-fiber Q}. If 5 is a 
LT-fiber, let S 1 + 1 be the LT-fiber obtained by adding 1 to each component of each vector 
in S. 

From Theorem 1, we know that there is a probability distribution (q(s)) on the state 
space S G such that 

q(s') = ^2 q(s)card({x G A : V(s, x) G {s' , s' + l}})/4, s' G S G ; (12) 

seS G 

D(G) = q(s)card({x G A : V(s,x) £S G })/4. (13) 

se5 G 

Let (q(S)) be the probability distribution on S G \U such that 

q(S) = J2q(s), SeS G \U 

s£S 

Using properties (a)-(b) together with ( fT2l) ( fTBT) . one can show that 

g (5") = ^ q(S)card({x e {0,1} :U(S,x)e{S',S' + l}})/2, S'eS G \U;(U) 
ses G \n 

D{G) = q{S)card({x e {0,1} :U{S,x) ^S G \U})/2. (15) 

ses G \n 

We determined that: 



U(S l ,0) = 


S 2 


U{S U 1) = 


s 2 


U(S 2 ,0) = 


s. 


U(S 2 ,1) = 


s 3 


U(S 3 ,0) = 


Si 


U(S 3 ,1) = 


s. 


U(S 4 ,0) = 


Sq 


U(S 4 , 1) = 


s 5 + i 


U(S 5 ,0) = 


s 2 


U(S 5 ,1) = 


s 3 


U{S 6 ,0) = 


Ss 


U(S 6 ,l) = 


S 7 + l 


U(S 7 ,0) = 


s 9 


U(S 7 ,1) = 


Sio 


U(S 8 ,0) = 


Ss 


U(S 8 ,1) = 


S 7 + l 


U{S 9 ,0) = 


Sn 


U(S 9 ,1) = 


s 5 + i 


U(S w ,0) = 


s. 


U(S W ,1) = 


Si 2 


U(S 1U 0) = 


Se, 


U(S U ,1) = 


513 + 1 


U(S 12 ,0) = 


S 6 


U(S 12 ,1) = 


5 14 + 1 


U(S 13 ,0) = 


S 9 


U(S 13 ,1) = 


Sl5 


U(S u ,0) = 


s. 


U(S U ,1) = 


Sl6 


U(S 15 ,0) = 


s 9 


U(S 15 ,1) = 


Sio 


U(S 16 ,0) = 


Sn 


U(S 16 ,1) = 


Sio 



s 



There is only one probability distribution satisfying (JHJ). It is: 



P(Si) 


= 


P{S 2 ) 


= 99/1809 


P{S 3 ) 


= 212/1809 


P(S 4 ) 


= 268/1809 


P(S 5 ) 


= 198/1809 


P(S 6 ) 


= 194/1809 


P(S 7 ) 


= 194/1809 


P(Ss) 


= 194/1809 


PiSg) 


= 128/1809 


P{Sio) 


"i i o /-i ar\r\ 

= 112/1809 


P( S n) 


= 64/1809 




= 56/1809 


P(S 13 ) 


= 32/1809 


P(S U ) 


= 28/1809 


P(Si 5 ) 


= 16/1809 


P{Si e ) 


= 14/1809 



From equation (fl5l) . 

D(G) = q(S i )/2 + q(S 6 )/2 + q(S s )/2 + q(S 9 )/2 + q(S u )/2 + q{S 12 )/2 
= 452/1809 
= 0.2499- ■■ 
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